Machine Learning as a Source of Insight into Universal Grammar

It is widely believed that the scientific enterprise of theoretical linguistics and the engineering of language applications are separate endeavors with little for their techniques and results to contribute to each other at the moment. In this paper, we explore the possibility that machine learning approaches to natural-language processing (NLP) being developed in engineering-oriented computational linguistics (CL) may be able to provide specific scientific insights into the nature of human language. We argue that, in principle, machine learning (ML) results could inform basic debates about language, in one area at least, and that in practice, existing results may offer initial tentative support for this prospect. A basic conundrum of modern linguistic research is the question of how natural languages can be acquired. It is uncontroversial that the learning of a natural language (or of anything else) requires some assumptions concerning the structure of the phenomena being acquired. So, for example, consider a computer program that is designed to learn to classify emails as spam or non-spam. One could assume that spam is identified by the magnitude of a weighted average over a set of indicator features fi, each of which specifies whether a given word wi appears in the email. We could then use a corpus of email messages, each marked as to their spam status, to train the weights appropriately. Such a learning approach (essentially perceptron learning over the features) assumes (erroneously, of course, but often effectively) that the phenomenon being investigated, spam status, is characterizable independently of any properties of the email but those manifest in the word-existence features, and that their impact on spam status is strictly linear. Alternatively, some other set of features, method for computing their combination, and so forth might be used, reflecting other assumptions about the nature of the phenomenon being investigated. In the field of machine learning, the prior structure imputed to the phenomenon is referred to as the model, which allows for variation in a set of parameters that may be of different sorts—continuous or discrete, fixed or combinatorial. Learning algorithms are procedures for setting these parameters on the basis of samples (observations) of the phenomenon being acquired. The success of the algorithm relative to the observations can be verified by testing to see if it generalizes correctly to unseen instances of the phenomenon. The role of the model and algorithm is to provide a learning bias.1 In linguistics, this prior structure is sometimes referred to as universal grammar (UG), the innate (that is experientially prior) aspect of the human language endowment that allows (biases) language acquisition. In the sequel, we will uniformly use the term learning bias for this model or universal grammar aspect.